Reference chains for surviving live-heap samples via a bounded manual heap walk (PROF-15341) - #644

The fixed defaults (budget=1000, ttl=60s, framecap=65536) are tuned for a
small heap. On a 4 GiB heap the BFS explores only ~60k edges before
the 60s TTL expires, abandoning every search without meaningful
progress.

Add ReferenceChainTracker::autoTuneDefaults() which scales unset
defaults from the process's max heap size and available processor
count, called from start() after LivenessTracker resolves max heap
but before the frontier table is constructed. Only overrides
defaults the operator did not set explicitly (tracked by
_reference_chains_tuned_mask in Arguments).

Tuning logic:
  budget:       sqrt(heap_mib / 512) * default (4GiB -> 2x, 16GiB -> 4x)
  firstpass:    10x budget (one-shot root enumeration can afford more)
  ttl:          (heap_mib / 512) * default, clamped [60s, 30min]
  framecap:     budget-scaled (larger budget -> larger frontier)
  pausetarget: 1 + (nprocs-1)/3 * default, capped 50ms
  painbudget:   1 + (nprocs-1)/4 * default, capped 5%

hop_cap is left alone (bounds chain depth, not breadth).

…scalation

The TTL was a time-based bound, but the search's completion is a
graph-size problem. On a 4 GiB heap, the BFS needs ~2000 passes at
2000 edges/pass — the TTL (auto-tuned to 840s) kills the
search at pass 336, frontier only half full, making progress but
not done. Then the pain budget drains, restarts, and repeats —
burning CPU forever without producing chains.

Proposal 1 — progress-based termination:
Replace the TTL abandonment with a no-progress detector.
Abandon only after NO_PROGRESS_PASS_LIMIT (30) consecutive passes
with zero new frontier entries admitted (genuinely stuck, not just
slow). A large heap takes more passes simply because there are
more objects — that is not 'stuck'. Suppressed when
isUrgent() (secondsToOOM() < OOM_URGENT_THRESHOLD_S): the search
must complete before the app OOMs, bounded only by the per-pass
STW pause SLO.

Proposal 2 — urgency-driven budget escalation:
When isUrgent(), raise the per-pass pause target to
URGENT_PAUSE_TARGET_MS (50ms) and tighten cadence to
URGENT_CADENCE_NS (10ms) so passes run back-to-back.
Also raise _budget 4x so the PID controller can let each pass
explore more edges. The total STW time is the same, but the
search converges in fewer, longer passes.

The _pass_deadline_ns and updatePacing() now use
_effective_pause_target_ms instead of the static _pause_target_ms.
resetSearchStateForTest() resets _passes_since_last_progress.
Updated the TTL unit test to verify the new wiring.

Proposes replacing the whole-graph forward BFS with a pruned
forward BFS that treats leaked objects as leaves and
terminates early when all leaked candidates are found.

Key design decisions (from review):
- Marker tag must be negative (class-tag range) to avoid
  collision with frontier tags (positive jlong from
  _next_tag).
- Do not return JVMTI_VISIT_ABORT when pruning --
  that aborts the entire FollowReferences walk,
  not just the candidate subtree. Simply skip
  the admitObject() call instead.
- Pre-tag candidates in threadLoop()'s first iteration, not
  in start() -- keeps everything on the BFS
  thread.
- Cleanup via GetObjectsWithTags(MARKER_TAG) -- don't
  re-resolve jweak references (candidate may have died).
- resetSearchStateForTest() must reset the candidate set.
- Cost claim softened to 10-100x (worst case is same as
  whole-graph BFS if candidates are deep).

This reverts commit bd81ff5.

Replace the whole-graph forward BFS with a pruned forward
BFS that treats leaked objects as leaves and terminates
early when all leaked candidates are found.

Phase 1: Candidate state fields on ReferenceChainTracker
- MARKER_TAG_BASE (negative, disjoint from frontier tags)
- _candidate_count, _candidate_found_bits, _candidate_tags[],
  _candidate_frontier_tags[]
- Init in start(), reset in resetSearchStateForTest()

Phase 2: Pre-tag candidates in threadLoop()
- Before first runPass(), call selectLeakCandidates() and
  SetTag(obj, MARKER_TAG_BASE - i) for each candidate.
- Gated on gcGenerationsEnabled() (no-op without it).

Phase 3: Prune heapReferenceCallback() at candidates
- Check *tag_ptr <= MARKER_TAG_BASE (marker tag, negative).
- Record chain link via FrontierTable::insert() with the marker
  tag as the frontier key (JVMTI SetTag overwrites, so the
  marker tag doubles as the buildChainEvent() lookup key).
- Skip admitObject() (don't enqueue children).
- Do NOT return JVMTI_VISIT_ABORT (aborts entire walk).

Phase 4: Early termination
- After each pass, if popcount(_candidate_found_bits) ==
  _candidate_count, transition to COMPLETED.

Phase 5: Cleanup
- On completion/abandonment, GetObjectsWithTags for each marker
  tag, SetTag(obj, 0), DeleteLocalRef, Deallocate.
- Reset candidate state.

Phase 7: pollWatchedTargets() canary path
- When getTag() returns a marker tag (negative), use it directly
  for buildChainEvent() lookup instead of the normal
  frontier tag path.

With canary pruning, total STW is <20ms per 60s recording
in practice. Use that wiggle room:

- Remove cadence sleep between passes when canary
  search is active (candidates still to find).
  shouldRunPass() returns true immediately for an active
  canary search, bypassing the cadence gate.
  The PID controller self-regulates per-pass
  STW pause.

- Frontier cap hit no longer abandons the search
  immediately. The search stays RUNNING:
  existing frontier entries may still lead to
  candidates, and the no-progress detector handles
  abandonment. The frontier cap is now a memory bound,
  not a correctness bound.

Updated tests:
- FrontierCapAbandonsSearchAndReleasesTags ->
  FrontierCapHitDoesNotAbandonImmediately (verifies search
  stays RUNNING after cap hit).
- ReleaseSearchTagsFailureBlocksTagReuseUntilItSucceeds
  updated to use no-progress abandonment instead
  of frontier cap (which no longer abandons).
- ReferenceChainsTestAccessor::reset() now resets
  _passes_since_last_progress and candidate fields.

…ve tags

FrontierTable::lookup() rejects tags <= 0 (it uses tag-1
as array index). Marker tags are negative
(MARKER_TAG_BASE - i = -(1LL << 62)), so
buildChainEvent(marker_tag) could never find the
candidate's entry -- no chains were produced.

Fix: store the per-candidate chain link (parent_tag,
referrer_klass, depth) in dedicated arrays
(_candidate_parent_tags[], _candidate_referrer_klasses[],
_candidate_depths[]) at pruning time. Add
buildCanaryChainEvent(candidate_idx) that walks
parent_tag (positive, so lookup() works) through the
frontier table and prepends the candidate's
own referrer_klass. pollWatchedTargets() uses
buildCanaryChainEvent() for canary candidates
(tag <= MARKER_TAG_BASE) and buildChainEvent()
for normal candidates.

REFERENCE_CHAIN_CANDIDATE_COUNT: number of leaked candidates
pre-tagged before the walk.
REFERENCE_CHAIN_CANDIDATES_FOUND: number of candidates
found so far in the current search (popcount
of _candidate_found_bits).

Set in threadLoop() after pre-tagging and
in runPass() on canary early termination.

The canary pruning check (*tag_ptr <= MARKER_TAG_BASE) was placed
AFTER the *tag_ptr < 0 class-tag check, which
returns 0 for any negative tag. Since marker
tags are negative (MARKER_TAG_BASE - i = -(1LL << 62)),
they hit the class-tag path first and never reach
the canary pruning code — candidates were
never detected, so no chains were produced.

Fix: move the canary pruning check BEFORE the
*tag_ptr < 0 class-tag check. Also fix
static member function access (use
ctx->tracker->_candidate_count instead of _candidate_count).

In heapReferenceCallback() pruning, _candidate_referrer_klasses was
set to parent.referrer_klass (the referrer's
klass), not the candidate's own klass.
The chain would have the parent's klass twice
and the candidate's klass missing.

Fix: resolve class_tag (JVMTI's klass tag
of the referent = the candidate) via classTags()->resolve()
and store that as the candidate's own klass.

When a leaked candidate is directly reachable from a GC root
(static field, JNI global, thread stack), the referrer_tag_ptr
is nullptr and rtag = 0. The 'if (rtag > 0)' check
failed, so the chain link was never recorded --
candidates_found stayed 0 and no chains were produced.

Fix: add an else branch for root-referenced candidates.
Record with parent_tag=0, depth=1, and
root_kind=reference_kind (the JVMTI
root kind for this reference). Also fix buildCanaryChainEvent
to handle parent_tag==0 (chain is just [candidate_klass]
with root_kind from the frontier entry).

… iteration

The pre-tagging was gated on !_search_started, so it only ran
once. If the search started before any candidates
existed (generations just enabled, no positive-slope klass yet),
candidates would appear later but pre-tagging
would never run -- _search_started was
already true.

Fix: remove the !_search_started gate. Pre-tag
whenever _candidate_count == 0 and candidates are available
from selectLeakCandidates(). This handles the
case where candidates appear after the search has already
started.

Logs why a klass fails the qualifying-growth check:
INSUFFICIENT_FILL (ring not full enough), SLOPE_TOO_SMALL
(mean growth below threshold), or floor_rising=false (minimum not
rising). This helps diagnose why a leak with a clear
upward population trend isn't being flagged as a
candidate.

The thirds approach (comparing earliest third vs recent third of the
ring) only uses 2/3 of the data and is noisy for
oscillating-but-growing trends. A klass that grows
steadily over 30+ GC epochs but oscillates within each
30-sample window can show a negative third-vs-third
slope even though the overall trend is clearly up.

Replace ringThirdsStats with a full-window least-squares
linear regression (y = a + b*x, x = sample position 0..fill-1).
This uses all samples and produces a proper slope
that matches what a dashboard graph shows.

Updated SecondsToOOMTest.RisingFloorProjectsExpectedSeconds:
time_delta changed from 7s (thirds: 8-1) to 9s (full window: 9-0).
Expected secondsToOOM changed from 10.0 to 9.0.

Add TEST_LOG to:
- recordHeapFloorSample: logs used bytes + timestamp for each sample
- onGC: logs when heap floor is recorded + GC epoch
- secondsToOOM: logs ring fill, bytes_delta, time_delta, means,
  and why it returns -1 (insufficient fill, not rising, etc.)
- heapFloorRising: logs mean/floor values + why it returns false
- selectLeakCandidates: logs summary (candidate count, required hysteresis,
  heapFloorRising state)
- hasLeakSignal: logs final verdict (secondsToOOM + candidates)

This gives full visibility into why the leak detector
isn't firing, in one deployment.

…or ring

The heap floor ring shows fill=0 despite onGC recording a sample.
Added TEST_LOG to recordHeapFloorSample to see whether it's
entered and whether _heap_floor_recording_disabled_for_test
is incorrectly true on the hotdog.

On JDK 26 with generational ZGC, _used_at_last_gc (read
from CollectedHeap via VMStructs) reports the committed
heap size, not the used bytes. The heap floor
would appear flat (7.5GB constant) even as the real
used bytes grow (2.19GB and rising).

Fix: in resolvePostGcHeapUsage(), when UseZGC is
enabled and hotspot_version >= 26, fall back to
HeapUsage::get(false)._used (the JMX path) instead of
_used_at_last_gc. Scoped to ZGC on JDK 26+ only —
other GC/JDK combinations keep the existing
_used_at_last_gc path.

VMFlag::find is a VMStructs table walk — don't call it on
every GC. Cache the result (UseZGC && hotspot_version >= 26)
as _is_zgc_jdk26_plus in initialize(), read it in
resolvePostGcHeapUsage().

LEAK_GROWTH_ABS_MIN was 5 (minimum 5 instances/epoch slope).
For large-population klasses (e.g. byte[] with thousands of
instances), a slope of 0.5 instances/epoch is
significant (~5% growth) but far below 5. The absolute
floor of 5 was too high, rejecting real leaks
in large-population klasses.

Lowered to 1 so only klasses with very small populations
(< 7 instances) need an absolute floor. This still
rejects single-instance oscillation noise while allowing
real leaks in large-population klasses.

Also: cache ZGC JDK26+ check in initialize() with debug logging.

…n't pre-tagged

selectLeakCandidates() was reading cached_slope directly without calling
hasQualifyingGrowth(), so consecutive_positive was not
updated for the current scan. The cached_slope could be stale
(set by a previous push with a negative slope), causing
selectLeakCandidates to skip the klass even though the
current trend is positive.

Fix: call hasQualifyingGrowth(entry) in selectLeakCandidates()
so consecutive_positive is updated and the slope is fresh.
Made hasQualifyingGrowth() take const ref (cached_slope is mutable).

Instead of pre-tagging candidate objects with negative marker tags
(MARKER_TAG_BASE - i), match by class_tag in heapReferenceCallback().
This avoids SetTag entirely: the walk visits every object
normally, and when it encounters one whose class matches
a candidate klass ID, it records the chain link and
skips expanding that object's children.

- _candidate_tags[i] now holds the klass ID (not marker tag)
- _candidate_frontier_tags[i] holds the frontier tag assigned
  by nextTag() when pruning
- buildCanaryChainEvent() uses _candidate_frontier_tags[i] as
  both the frontier table key and the target_tag
- pollWatchedTargets() canary path checks tag > 0 (frontier tag from
  heapReferenceCallback) instead of tag <= MARKER_TAG_BASE

Tests need updating: PollWatchedTargetsTest expects the old
marker-tag approach. 3 tests fail.

Tests now set _candidate_frontier_tags via
setCandidateFrontierTagForTest() so
buildCanaryChainEvent() can reconstruct the chain.
Added setCandidateFrontierTagForTest() test accessor.